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Sandra  P.  Marshall 
Department  of  Psychology 

Center  for  Research  in  Mathematics  and  Science  Education 
San  Diego  State  University 
San  Diego,  CA  92182 

The  purpose  of  this  paper  is  to  explore  some  of  the 
issues  involved  in  assessing  cognitive  development.  The 
paper  is  divided  into  four  sections.  In  the  first  section  I 
examine  the  need  for  theories  of  testing  and  for  procedures, 
that  go  beyond  those  currently  in  use.  In  the  second 
section  I  propose  an  assessment  model  based  upon  schema 
knowledge,  and  in  the  third  I  offer  data  from  an  intelligent 
computer-assisted  instruction  (ICAI)  system  based  on 
schemas.  In  the  fourth  section  I  summarize  my  findings  and 
suggest  additional  questions  that  need  to  be  addressed  by 
further  research. 

The  Need  for  New  Theories 

A  major  focus  in  current  educational  and  psychological 
research  is  on  higher-order  thinking  skills.  While  these 
are,  in  a  sense,  just  "buzz  words",  they  suggest  an  interest 
in  processes  that  go  beyond  retrieval  of  facts,  algorithms, 
concepts,  or  rules.  The  emphasis  is  upon  how  an  individual 
puts  together  the  knowledge  at  his  or  her  disposal  and  upon 
the  product  that  emerges  as  a  result  of  that  synthesis.  The 
measurement  question  that  accompanies  this  new  emphasis  is: 
How  can  we  assess  this  type  of  cognition  and  performance? 

Elsewhere  I  have  outlined  two  approaches  that  may  be 
useful  in  the  evaluation  of  schema  knowledge,  item  response 
theory  (IRT)  and  statistical  graph  theory,  and  I  explored 
there  some  of  these  measurement  issues  (cf.,  Marshall,  in 
press).  I  think  that  IRT  and  statistical  graph  theory  are 
viable  means  for  modeling  schema  assessment,  but  they  are 
still  far  from  being  usable  now.  Both  approaches  contain 
important  psychometric  issues  to  be  resolved.  In  the 
remainder  of  this  paper,  I  adopt  principally  the  graph 
theoretical  perspective,  but  I  do  not  attempt  to  deal  here 
with  some  of  the  sampling  and  estimation  problems  I  discuss 
in  the  earlier  paper.  My  purpose  in  the  present  paper  is  to 
show  how  we  can  begin  to  assess  schema  knowledge  even  though 
a  full  testing  model  is  not  yet  available. 


An  Assessment  Model  for  Schema  Knowledge 

In  this  section  I  address  the  issue  of  assessment  of  a 
particular  kind  of  knowledge,  generally  called  schema 


knowledge.  My  interest  in  developing  new  theories  of 
psychological  testing  stems  from  my  research  in  knowledge 
and  how  individuals  store,  manipulate,  and  retrieve  it  from 
long-term  memory.  For  several  years  I  focused  on  the 
formation  of  schemas,  and  I  have  frequently  considered  the 
problem  of  how  to  evaluate  for  an  individual  whether  these 
knowledge  structures  exist,  whether  they  are  partially 
constructed,  whether  elements  of  them  are  incorrect  or 
simply  absent,  and  whether  they  might  be  present  but  simply 
not  used  on  a  particular  task.  None  of  these  is  easily 
answered . 

To  make  matters  worse,  the  schema  as  a  psychological 
construct  has  been  poorly  defined  in  my  own  area  of 
cognitive  psychology.  Some  researchers  approach  it  from  an 
architectural  point  of  view  —  that  is,  they  are  concerned 
primarily  with  the  schema  as  a  structural  feature  of  long¬ 
term  memory,  usually  specifying  it  as  a  set  of  nodes  and 
arcs  connecting  those  nodes.  Other  researchers  emphasize 
the  nature  of  the  knowledge  that  is  contained  in  the  schema, 
that  is,  the  particular  content  features. 

If  we  are  to  assess  schema  knowledge  adequately,  we 
need  to  encompass  both  perspectives.  Our  assessment  must 
take  into  account  both  how  we  believe  knowledge  to  be  linked 
(i.e.,  the  structural  properties)  and  what  we  believe  to  be 
stored  in  the  schema  (i.e.,  the  content  properties).  We 
need  to  estimate  not  only  the  degree  of  connectivity  and  the 
size  of  the  network 

represented  by  the  schema  but  also  the  different  types  of 
knowledge  that  have  been  accommodated  by  it. 

As  I  said,  these  are  not  trivial  questions.  There  is  a 
great  deal  of  psychological  theory  development  to  be  done. 
We  don't  yet  have  a  hard  and  fast  conception  of  what  a 
schema  is.  Nevertheless,  I  think  we  can  begin  to  make 
progress  by  formulating  a  working  definition,  hypothesizing 
the  necessary  components,  and  testing  those  hypotheses.  As 
we  test  the  hypotheses,  we  of  necessity  become  involved  in 
building  a  new  type  of  psychological  testing.  In  the 
remainder  of  this  paper  I  describe  my  attempts  at  modeling 
schema  knowledge  and  evaluating  the  model  with  respect  to 
students'  understanding  of  arithmetic  story  problems. 


The  Nature  of  a  Schema 


A  brief  review  of  my  previous  work  may  be  helpful.  I 
have  been  working  with  a  particular  theory  of  schema 
knowledge  and  its  formation  for  about  two  years.  In  this 
theory  a  schema  consists  of  four  distinct  types  of 
knowledge:  feature  recognition,  constraint  mapping, 
planning/goal-setting,  and  execution.  These  together 


comprise  a  network  of  schema  knowledge,  with  the  four 
components  being  the  nodes  and  having  links  among  them. 
Moreover,  I  assume  that  these  four  components  are  themselves 
structured  as  networks  of  nodes,  with  various  linkages 
connecting  these  nodes. 

The  feature  recognition  component  contains  declarative 
knowledge  about  the  schema,  including  a  general  description. 
All  of  the  pertinent  features  that  describe  the  content  of 
the  schema  reside  here.  The  constraint  mapping  component 
houses  the  set  of  rules  that  govern  the  instantiation  of  the 
schema.  Certain  conditions  must  be  met  if  the  schema  to  be 
used.  The  first  component  dealt  with  recognition  of  the 
basic  features.  This  second  one  tests  whether  or  not  a 
sufficient  set  of  features  are  present. 

The  third  component  of  schema  knowledge  has  to  do  ”ith 
making  plans  for  implementing  the  schema,  making  estimates 
about  its  outcome,  and  drawing  appropriate  inferences 
related  to  its  use.  The  mechanisms  for  setting  goals  and 
subgoals  reside  here.  Finally,  the  fourth  component, 
execution,  encompasses  the  rules  and  procedures  by  which  the 
schema  is  formally  implemented. 

A  successful  call  to  a  particular  schema  means  that 
the  individual  recognizes  a  situation  (feature  recognition), 
determines  if  critical  circumstances  are  present  (constraint 
mapping),  formulates  appropriate  plans  for  using  the  schema 
(planning/goal-setting),  and  carries  out  those  plans 
(execution).  For  the  schema  to  be  a  useful  knowledge 
structure,  these  four  components  must  be  connected. 

Each  of  the  four  types  of  knowledge  forms  a  network. 
This  network  may  or  may  not  be  connected  to  the  other  types 
of  knowledge,  depending  upon  the  level  of  schema 
construction  that  has  taken  place.  A  fully  developed  schema 
would  have  many  links  among  components  as  well  as  within 
them. 

How  are  we  to  measure  the  development  of  a  schema? 
Clearly  one  way  is  to  examine  its  connectivity.  One  of  the 
most  important  features  of  a  schema  is  the  degree  to  which 
its  constituent  pieces  are  connected  to  each  other.  In  the 
language  of  cognitive  psychology,  this  is  an  issue  of 
activation.  If  an  individual  retrieves  one  particular  piece 
of  information,  how  many  other  pieces  are  also  automatically 
retrieved  and  available  for  processing? 

Consider  the  simple  case  represented  in  Figure  1.  Here 
there  are  only  two  components,  X  and  Y.  Nodes  A,  B,  and  C 
are  elements  in  one  component;  nodes  E,  F,  and  G  are 
elements  of  the  second.  In  Figure  1(a),  these  two 
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components  are  unconnected,  although  each  component  itself 
is  fully  linked  (i.e.,  each  node  is  connected  to  every  other 
node).  Given  a  prompt  to  node  A,  an  individual  having  this 
knowledge  structure  would  be  able  to  access  all  the 
information  contained  in  component  X  but  would  be  unable  to 
access  the  nodes  of  Y.  Similarly,  a  call  to  E  would  result 
in  activation  of  Y  while  leaving  X  unreachable. 


In  Figure  1(b)  the  two  components  are  linked  through 
nodes  A  and  E.  Now  a  call  to  X  would  result  in  access  to 
the  nodes  in  both  components.  There  are  many  ways  that  X 
and  Y  could  be  linked,  ranging  from  a  single  connection  as 
in  Figure  1(b)  to  all  possible  connections,  as  shown  in 
Figure  1(c).  For  simplicity,  if  all  the  elements  in  one 
component  are  linked  together,  all  the  elements  in  a  second 
component  are  linked  as  well,  and  if  there  exists  one  or 
more  paths  between  the  components,  the  entire  components  are 
considered  to  be  connected  and  can  be  represented  as  in 
Figure  1(d).  The  single  arc  between  X  and  Y  carries  a 
weight  indicating  whether  the  link  between  the  components  is 
weak  (as  in  Figure  lb)  or  strong  (as  in  Figure  lc). 
Estimating  the  strength  of  this  link  will  be  an  important 
task  for  those  who  wish  to  assess  schema  knowledge.  This 
aspect  of  schema  assessment  takes  into  account  the 
architectural  structure  of  the  schema. 


A  second  way  to  measure  schema  knowledge  is  to  focus  on 
the  types  of  knowledge  that  are  linked  together.  That  is, 
our  target  knowledge  may  have  multiple  links.  Just  knowing 
the  number  of  links  does  not  tell  us  all  we  want  to  know 
about  the  connectivity.  The  links  may  be  connections  to 
several  different  kinds  of  knowledge  or  they  may  be  pointers 
instead  to  many  instances  of  a  single  kind.  Figure  2 
contains  two  graphs  illustrating  different  kinds  of 
linkages.  In  part  (a),  node  A  has  six  arcs  leading  to  six 
different  nodes.  Each  of  these  is  a  terminal  node,  having 
no  other  connections.  Typically,  this  configuration 
corresponds  to  a  graph  of  a  concept  with  various  defining 
features  linked  to  it  (e.g.,  a  bird  has  feathers,  eats 
seeds,  makes  a  chirping  sound).  The  features  are  not 
themselves  linked  together.  All  of  the  nodes  in  Figure  2a 
represent  the  same  type  of  knowledge.  A  different 
configuration  is  shown  in  Figure  2(b).  Here,  node  A  has 
four  links,  three  within  the  component  and  one  to  a  node  in 
another  component.  This  pattern  suggests  a  better 
integration  of  knowledge. 


Schema  Identification 


Schema  measurement  requires  specification  of  the 
schemas  that  are  of  importance  in  the  domain.  Let  me  give 
an  example.  In  my  own  research  I  study  how  individuals 


learn  to  understand  and  solve  arithmetic  story  problems.  To 
determine  what  the  primary  schemas  are  for  this  domain  one 
asks  what  one  expects  of  a  student  who  has  mastery  of  the 
domain.  What  would  we  like  for  the  student  to  be  able  to 
do?  What  should  he  or  she  demonstrate  to  exhibit 
competence?  For  the  case  of  arithmetic  story  problems  it  is 
not  enough  that  the  students  have  mastery  of  operations. 
What  we  expect  of  them  is  the  ability  to  understand  the 
situation  expressed  as  a  story  or  word  problem  and  the 
skills  to  extract  the  necessary  information  from  the 
situation.  These  together  with  the  means  to  carry  out  the 
computations  indicate  mastery. 

In  addition  to  specifying  the  schemas,  we  must  also, 
specify  for  each  one  the  elements  in  the  requisite  four 
components  of  feature  recognition,  constraint  mapping, 
planning/goal-setting,  and  execution.  Thus,  we  construct 
an  ideal  representation  of  the  schema  and  specifies  the 
necessary  links  that  should  exist  among  elements  (e.g., 
nodes)  and  between  components  (e.g.,  subgraphs).  One  can 
think  of  the  entire  domain  as  a  very  large  graph.  Within 
this  large  graph  are  subgraphs  corresponding  to  the 
individual  schemas.  Within  these  subgraphs  are  smaller 
subgraphs  corresponding  to  the  components  of  the  schemas. 
Finally,  within  the  subgraphs  are  even  smaller  subgraphs 
consisting  of  the  elements  that  form  a  single  schema 
component.  Figure  3  is  an  example  of  a  very  simple  domain 
having  three  schemas. 

As  shown  in  Figure  3,  several  levels  of  connectivity 
will  exist.  First,  the  schemas  themselves  ought  to  be 
linked  together  to  form  a  cohesive  knowledge  structure  about 
the  domain.  There  will  almost  certainly  be  common  elements 
within  two  or  more  schemas  and  one  expects  that  the  larger 
subgraphs  would  be  linked  together. 

Second,  within  each  schema  the  four  components  should 
be  connected.  For  a  schema  to  be  successfully  instantiated, 
all  four  parts  of  schema  knowledge  are  necessary.  The 
components  cannot  function  in  isolation. 

Finally,  within  any  component  one  expects  to  have  a 
well-connected  subgraph  of  nodes.  The  elements  at  this 
level  have  the  most  similarity  one  to  another  and  have  the 
most  direct  association.  Many  paths  among  nodes  would  be 
expected  rather  than  a  single  one.  Similarly,  one  expects 
to  have  a  path  from  any  node  in  the  component  to  all  other 
nodes  (but  not  necessarily  a  direct  link  from  each  node  to 
all  others).  The  component  level  of  the  network  is  not 
shown  in  Figure  3.  Each  of  the  nodes  labeled  D,  C,  P,  and  E 
corresponds  to  a  subgraph  of  individual  elements  linked 
together  to  form  the  component. 


Given  this  model  of  a  domain,  we  desire  now  to  have  a 
testing  model  that  reflects  it.  The  goal  is  a  model  that 
takes  into  account  the  specific  nodes  that  are  required  of  a 
schema  and  the  ways  in  which  those  nodes  are  related  to  each 
other . 

Several  obvious  points  can  be  made.  First,  if  we  only 
want  to  test  the  individual  elements  within  any  component  of 
schema  knowledge,  we  can  do  so  easily  and  can  use  the 
student's  response  to  estimate  the  absence  or  presence  of 
the  element  we  test.  We  may  decide  that  one  test  item  per 
element  is  sufficient  or  we  may  wish  to  use  multiple  items. 
Similarly,  we  can  evaluate  whether  the  student  has  two 
elements  of  knowledge  within  the  same  component  by 
constructing  a  test  item  that  requires  both.  In  many  cases 
we  may  assume  connectivity  if  the  student  can  respond  to  the 
item  with  the  appropriate  elements.  So,  for  example,  if  we 
are  assessing  the  descriptive  elements  of  a  particular 
schema,  we  might  ask  a  student  to  provide  a  general 
description  of  the  relation  for  which  the  schema  has  been 
constructed.  Each  distinct  feature  given  in  the  student's 
response  can  be  treated  as  an  element  of  the  descriptive 
component  and  can  be  considered  to  be  linked  to  all  other 
features  in  the  response. 

When  we  move  up  a  level  of  complexity  to  look  at  the 
components  and  their  degree  of  connection,  we  have  greater 
difficulty  in  constructing  test  items.  The  difficulty  is 
that  we  must  have  test  items  that  represent  two  different 
dimensions  of  schema  knowledge.  For  example,  suppose  we 
uesxte  to  assess  whether  an  individual  has  linked  components 
of  declarative  and  planning  knowledge.  Our  test  item  must 
call  for  demonstration  that  the  declarative  component  is 
already  established,  for  evidence  of  planning  knowledge 
relevant  to  the  schema,  and  for  the  application  of  the 
second  to  the  first.  The  item  needs  to  be  well  constructed. 
In  particular,  if  the  student  answers  the  item  incorrectly, 
we  want  to  be  able  to  ascertain  from  the  response  whether 
the  error  arose  in  the  declarative  component,  in  the 
planning  component,  or  in  the  link  between  the  two. 

The  purpose  of  cognitive  assessment  as  it  is  presented 
here  is  to  estimate  how  fully  developed  and  how  completely 
related  are  the  components  of  schema  knowledge  for  an 
individual.  Each  test  item  contributes  to  our  estimate  of 
the  pieces  of  knowledge  and/or  to  their  connectivity.  In 
the  assessment  of  schema  knowledge,  not  all  test  items 
provide  the  same  degree  of  information  about  the  individual. 
Some  items  test  many  distinct  parts  of  the  schema  model  and 
hence  provide  multiple  estimates.  Some  test  only  a  single 


element  or  a  single  link.  Consequently,  the  items  may  need 
to  be  differentially  weighted. 

In  the  following  section  I  give  examples  of  different 
assessment  items  and  the  nature  of  the  diagnostic 
information  we  gather  from  these  items.  Profiles  are 
developed  for  a  small  number  of  students  to  demonstrate  how 
the  information  can  be  aggregated. 

Current  Implementations 

I  have  recently  developed  an  ICAI  system  designed  to 
teach  students  how  to  understand  the  relationships  expressed 
in  arithmetic  story  problems.  This  system  was  created 
around  five  specific  schemas  used  in  arithmetic.  Each  of 
the  four  knowledge  components  of  these  five  schemas  is 
explicitly  addressed  in  the  system,  and  we  have  a  number  of 
diagnostic  tasks  that  are  intended  to  evaluate  whether  or 
not  students  have  encoded  and  linked  together  the  necessary 
pieces  of  knowledge.  Details  about  the  five  relations 
taught  by  the  ICAI  system  may  be  found  elsewhere  (Marshall, 
Pribe,  &  Smith,  1987). 

A  necessary  part  of  the  system  is  the  assessment  of 
whether  or  not  a  student  has  acquired  the  requisite 
knowledge  at  a  particular  time.  The  most  common  method  of 
evaluation  of  ability  to  solve  arithmetic  story  problems  is 
to  present  a  set  of  problems  and  to  count  the  number  of 
problems  for  which  che  correct  numerical  solution  was 
obtained.  Such  an  approach  tells  us  little  about  schema 
formation  or  development  and  offers  almost  nothing  of 
diagnostic  value  about  which  components  of  the  schema  are 
weak  or  incomplete. 

It  is  important  to  say  here  that  we  never  ask  students 
to  perform  this  traditional  task  in  our  ICAI  system.  We 
focus  the  evaluation  questions  instead  upon  specific  aspects 
of  each  schema.  Below  I  briefly  describe  the  types  of 
instruction  we  give  and  the  questions  we  ask.  Details  about 
the  curriculum  and  the  particular  schemas  are  not  elaborated 
in  this  paper.  The  following  description  is  intended  to 
provide  a  general  framework  within  which  to  discuss  the 
types  of  evaluative  items  we  use  in  the  ICAI  system. 

Feature  Recognition.  The  feature  recognition  component 
of  a  schema  contains  elements  about  the  general  structure  of 
a  story  problem  appropriate  to  chat  schema.  Our  primary 
concern  here  is  not  the  assessment  of  each  element  within 
the  subgraph  "feature  recognition"  but  rather  the  assessment 
of  the  entire  subgraph.  Consequently,  our  tasks  for 
students  focus  on  their  ability  to  identify  and  discriminate 


among  the  situations  that  correspond  to  the  five  basic 
schemas  we  employ. 


As  part  of  the  feature  recognition  or  declarative 
portion  of  schema  knowledge  we  expect  a  student  to  encode  a 
verbal  label  for  the  relation  in  the  schema,  to  encode  a 
general  description  of  the  relation,  and  to  encode  a  simple 
pictograph  we  use  to  represent  the  relation.  Our 
instruction  focuses  upon  these  aspects. 

Two  tasks  form  the  foundation  of  our  assessment.  In 
one  of  them,  students  simply  examine  a  story  problem  and 
classify  it  according  to  situational  labels  (Change,  Group, 
Compare,  Vary,  and  Restate).  Incorrect  responses  by 
students  elicit  feedback  from  the  ICAI  system  that  points 
out  key  elements  of  feature  recognition  for  the  particular 
problem.  An  example  of  this  task  is  given  in  Figure  4. 

For  the  second  task  we  introduce  the  students  to 
diagrams  or  pictographs  for  each  situation  (see  Figure  5). 
Our  intention  here  is  for  the  students  to  associate  the 
features  of  the  situation  with  the  appropriate  diagram. 
Again,  students  are  presented  with  a  single  story  problem 
and  given  the  choice  of  diagram  from  the  five. 

Constraints  .  As  in  the  feature  recognition 
instruction,  the  constraint  instruction  has  two  aspects, 
verbal  and  diagrammatic.  The  verbal  constraints  are 
conditions  that  must  exist  for  the  schema  to  be 
appropriately  implemented.  An  example  of  a  verbal 
constraint  is  the  role  of  time  in  a  Change  relation.  The 
change  takes  place  over  time.  If  the  story  problem  does  not 
develop  over  two  different  periods  of  time,  it  cannot 
reflect  the  change  relation. 

The  diagrammatic  constraints  have  to  do  with  the  number 
of  different  parts  required  by  a  relation  and  the  ways  in 
which  they  fit  together.  Each  of  the  diagrams  or 
pictographs  have  from  three  to  five  slots  which  are  to  be 
filled  with  information  from  the  story  problem.  If  too  many 
of  the  slots  are  unfilled  or  if  there  are  more  features  of 
the  problem  than  slots  in  the  diagram,  the  diagram  is  an 
inappropriate  match  for  the  story  problem. 

Again,  there  are  two  basic  tasks.  In  the  first  task, 
students  compare  a  set  of  story  problems  and  attempt  to 
identify  the  situations.  Unlike  the  tasks  of  feature 
recognition,  this  task  presents  several  problems  at  once  to 
the  student.  Usually  the  student  is  given  five  problems, 
one  for  each  of  the  possible  schema  situations.  To  complete 
the  task  successfully,  the  student  not  only  compares  a 
single  problem  with  the  five  situations,  he  or  she  also  must 


I 


compare  the  problems  to  each  other.  For  example,  students 
frequently  believe  that  two  of  the  five  problems  demonstrate 
the  same  situation.  Since  the  task  is  to  identify  one 
problem  for  each  situation,  the  students  must  engage  in 
constraint  mapping  to  determine  which  of  the  problems 
actually  evidences  the  situation  and  which  does  not. 

In  the  second  task,  students  are  asked  to  use  the 
diagrams  and  to  map  relevant  pieces  of  the  story  problem 
into  the  parts  of  a  given  diagram  (see  Figure  6).  The 
students  do  not  select  the  diagram.  The  story  problem  and 
diagram  are  given,  and  the  student  merely  moves  pieces  of 
the  story  problem  into  the  diagram. 

Other  casks  explicitly  link  recognition  features  with 
constraints.  The  ICAI  system  presents  a  series  of  story 
problems  to  the  student.  For  each  story  problem  the  student 
must  select  the  appropriate  diagram  and  demonstrate  how  the 
story  problem  maps  into  it.  These  tasks  require  several 
responses  of  the  student.  First  the  student  identifies  the 
appropriate  diagram  as  in  Figure  5.  If  the  response  is 
incorrect,  relevant  feedback  is  provided  to  the  student  by 
the  computer.  Next  the  student  is  shown  the  larger  version 
of  the  appropriate  diagram  and  asked  to  map  the  story 
problem  (as  in  Figure  6).  These  tasks  require  the  students 
to  determine  whether  one  schema  situation  is  more 
appropriate  for  a  given  story  problem  than  another. 
Further,  they  force  the  student  to  test  for  himself  or 
herself  whether  the  constraints  of  the  schema  are  met  by  the 
particular  problem. 

Planning/Goal -Set ting  Questions .  One  of  our  objectives 
in  the  ICAI  system  is  to  help  the  student  understand  the 
problem  and  to  be  able  to  make  plans  about  solving  it.  We 
actually  do  not  assess  planning  alone.  Our  tasks  are 
designed  to  look  at  planning  and  goal-setting  with  respect 
to  feature  recognition  and  constraints. 

Two  levels  of  planning  are  possible.  First,  the 
student  has  to  anticipate  where  the  unknown  of  the  problem 
fits  into  the  general  schema  situation.  For  example,  the 
change  situation  has  three  parts:  the  beginning,  the  amount 
of  change,  and  the  result.  In  a  story  problem,  any  one  of 
these  three  parts  may  serve  as  the  unknown  in  the  problem. 
Choice  of  arithmetic  operation  depends  upon  which  part  of 
the  situation  is  unknown.  We  focus  on  this  issue  by  having 
the  student  first  examine  a  story  problem  and  its 
appropriate  blank  diagram  and  then  place  the  word  UNKNOWN  in 
the  appropriate  part  of  the  diagram.  (Later  tasks  relate 
the  operation  to  the  location  of  the  unknown.) 


Other  tasks  about  planning  are  relevant  for  story 
problems  with  multiple  steps.  Students  learn  to  recognize 
which  situation  governs  the  problem  and  which  subproblems 
must  be  solved  before  the  top-level  situation  can  be 
addressed.  We  use  several  tasks  here.  One  calls  for  the 
student  to  identify  in  a  diagram  which  parts  are  given  in 
the  problem,  which  are  partially  known  but  immediately 
solvable,  and  which  are  the  true  unknowns.  Figure  7 
contains  an  example  of  this  task. 

We  also  have  the  students  identify  the  primary  question 
asked  in  the  problem.  We  work  with  them  to  identify 
secondary  questions  and  the  situations  associated  with  them. 

Execution  Questions.  With  this  last  group  of  questions 
we  come  close  to  asking  the  students  to  solve  the  problems. 

However,  our  interest  is  not  in  whether  or  not  they  have 
mastered  arithmetic  algorithms.  We  are  concerned  with  their 
ability  to  formulate  the  appropriate  arithmetic  expressions 
to  reflect  the  problem  situation.  For  example,  students 
have  a  tendency  to  associate  a  particular  operation  with  a 
type  of  problem.  Take  a  Change  situation  as  described 
above.  Most  students  confronted  with  this  situation  want  to 
perform  an  addition.  This  is  not  necessarily  the 
appropriate  operation.  If  the  problem  contains  information 
about  the  starting  amount  and  the  change,  and  the  change  is 
an  increase,  then  addition  indeed  is  warranted.  If, 
however,  the  student  is  given  the  amount  of  change  and  the 
final  amount,  addition  will  not  solve  the  problem. 

Our  tasks  that  assess  execution  skills  ask  students  to 
identify  arithmetic  expressions  and  verbal  expressions  about 
arithmetic  operations.  Feedback  on  these  tasks  stresses  the 
importance  of  identifying  the  situation  correctly  and  then 
observing  which  part  of  the  problem  is  unknown. 

This  portion  of  the  ICAI  system  is  the  briefest.  Our 
current  emphasis  is  primarily  upon  the  recognition  features, 
the  constraints,  and  the  planning  components.  Consequently, 
our  assessment  of  the  execution  component  of  schema 
knowledge  is  not  as  strong  as  the  assessment  of  the  other 
three  components. 

Summary  of  the  ICAI  Evaluation.  I  have  described  the 
questions  used  in  the  ICAI  system  and  given  examples  to 
point  out  how  they  differ  from  traditional  questions 
assessing  ability  to  solve  arithmetic  story  problems.  A 
primary  asset  of  these  tasks  is  that  they  allow  us  to 
estimate  which  aspects  of  schema  knowledge  have  been  encoded 
in  memory  and  which  remain  to  be  formed.  Following  this 
series  of  tasks,  we  expect  to  determine  for  each  student  how 
fully  each  of  the  five  schema  knowledge  structures  has  been 


developed.  We  are  currently  analyzing  the  student  responses 
to  the  items  presented  by  the  computer. 

Four  student  profiles.  Below  I  describe  the  data  from 
four  students  and  how  we  estimate  their  schema  knowledge  for 
one  particular  relation,  the  Change  relation.  The  data  were 
gathered  over  five  hours  of  instruction.  All  instruction 
was  given  within  a  two-week  period.  The  students  were 
college  freshmen.  Preliminary  testing  of  these  students 
indicated  poor  problem-solving  skills.  On  a  test  of  10 
multi-step  story  problems,  the  four  students  (J,  N,  P,  and 
S)  answered  correctly  7,  5,  5,  and  7  problems  respectively. 
Table  1  contains  data  from  these  four  students  as  they 
worked  with  the  ICAI  system.  All  of  the  students  responded 
to  approximately  the  same  number  of  story  problems.  Their 
relative  performance  is  given  in  the  table. 

We  would  like  to  infer  from  student  performance  that 
individuals  who  are  more  successful  in  solving  problems  have 
greater  understanding  of  the  domain  than  students  who  are 
less  successful.  From  the  overall  performance  percentages 
we  can  only  conclude  that  some  students  answered  more  items 
correctly.  With  this  information  alone  we  cannot  tell 
whether  or  not  their  schema  knowledge  differs. 

In  the  lower  part  of  Table  1  are  profiles  of  these  four 
students,  based  upon  an  analysis  of  the  tasks  as  described 
above.  We  do  not  have  data  to  look  at  all  possible 
components  and  connections.  In  particular,  these  students 
did  not  complete  the  execution  portion  of  the  ICAI  system. 
Thus,  this  component  is  excluded  from  the  analysis.  Seond, 
only  Feature  Recognition  and  Constraints  are  assessed  as 
components.  While  we  provided  instruction  about  plann  ng 
elements,  our  Planning  tasks  themselves  explicitly  link  he 
planning  activities  either  to  feature  recognition  r 
constraints.  Thus,  the  profiles  have  some  of  the  componen  s 
and  some  of  the  connections.  Nonetheless,  it  is  useful  to 
compare  the  profiles  of  the  students. 

Students  J  and  N  primarily  had  difficulty  in  encoding 
feature  recognition  elements.  Most  of  their  errors  are  on 
these  items.  Since  feature  recognition  occurs  early  in  our 
instructional  system,  these  two  students  apparently  erred 
early,  learned  from  their  errors,  and  then  continued  with  a 
high  degree  of  accuracy.  Neither  of  these  students  had  a 
great  deal  of  difficulty  with  the  constraints  or  the 
planning  connections. 

Student  P  had  moderate  success  with  the  components  of 
feature  recognition  and  constraints  but  was  less  successful 
in  connecting  the  three  types  of  knowledge.  In  particular. 


Student  P  was  weak  in  linking  the  constraints  and  planning 
elements . 

Finally,  Student  S  was  generally  weaker  than  the  other 
students  in  all  areas.  The  feature  recognition  elements 
were  not  well  developed,  nor  were  the  constraints.  These 
components  in  turn  were  then  difficult  for  the  student  to 
link  through  the  planning  activities. 

What  do  the  profiles  tell  us?  First,  we  see 
differences  that  are  not  evident  from  percentage  of  correct 
score  alone.  Second,  we  can  use  the  information  from  the 
various  items  to  estimate  what  aspects  of  schema  knowledge 
are  not  well  formed.  If  we  intended  to  use  the  ICAI  system 
as  a  diagnostic  instrument  for  these  students,  we  would 
select  different  subsequent  instruction  for  them.  Student  J 
is  doing  well  and  should  progress  to  the  next  level  of 
instruction.  Student  N  is  performing  adequately  and  should 
perhaps  be  reevaluated  soon.  This  student  also  could  move 
on.  The  other  students  need  particular  and  different  types 
of  remediation.  Student  P  had  difficulty  with  the  visual 
parts  of  feature  recognition  and  constraint  components. 
This  student  may  need  special  instruction  about  the 
diagrams.  In  contrast.  Student  S  needs  assistance  in 
forming  connections.  The  degree  of  connectivity  displayed 
here  is  low  and  needs  to  be  improved. 

Finally,  we  can  use  the  profiles  to  evaluate  the 
instructional  system.  Which  parts  of  the  system  are  most 
effective?  Which  need  to  be  modified?  We  are  currently 
developing  profiles  for  the  remaining  16  students  who  have 
used  the  system.  We  anticipate  that  we  will  learn  more 
about  how  schema  knowledge  is  developed  when  we  have 
analyzed  the  full  set  of  data  for  the  20  students. 

Summary  and  Conclusions 

In  this  paper  I  have  presented  some  ideas  about  the 
types  of  knowledge  we  want  to  assess  in  individuals,  and  I 
have  outlined  a  theoretical  approach  for  making  that 
assessment.  Obviously,  we  still  have  a  long  way  to  go 
before  we  have  a  robust  theory  for  testing  schema  knowledge. 
Nevertheless,  I  am  encouraged  by  the  results  we  have 
obtained  thus  far.  We  are  able  to  parse  our  test  items  by 
the  hypothesized  components  of  schema  knowledge,  and  we  have 
achieved  a  profile  of  schema  knowledge  for  the  students  who 
interact  with  our  computer  system. 

Why  do  we  want  to  measure  schemas?  The  psychological 
theory  of  knowledge  acquisition  is  closely  tied  to  the 
primitive  theory  of  schema  assessment  presented  here.  Under 
the  assumption  that  the  components  of  schema  knowledge  have 


been  appropriately  identified,  we  can  now  equate 
understanding  to  schema  formation.  That  is,  understanding 
improves  as  an  individual  builds  the  components  of  schema 
knowledge  and  links  these  components  together.  The  recent 
stresses  on  higher  order  thinking  skills,  understanding, 
"owning  your  own  knowledge",  and  the  construction  of 
knowledge  by  the  individual  all  are  answered  in  some  degree 
by  schema  theory. 

The  perspective  developed  in  this  paper  is  not  a 
traditional  testing  approach.  First,  there  are  several 
attributes  to  be  measured,  because  schema  knowledge  has  many 
dimensions.  Second,  we  are  not  concerned  with  a  sample  from 
a  universe  of  items.  The  items  must  be  well  chosen  but  do 
not  necessarily  reflect  the  entire  domain.  It  is  likely 
that  only  a  few  items  will  be  possible  in  a  test  of  schema 
knowledge  because  responses  to  these  items  take  more  time 
than  responses  to  conventional  items.  Probably  assessment 
of  individuals  will  involve  different  items.  Once  a 
particular  linkage  is  demonstrated  by  the  student  response 
to  a  test  item,  we  have  no  need  to  assess  the  link  again. 
Failure  to  show  the  link,  however,  might  lead  to  a  set  of 
probing  questions  to  determine  whether  it  is  the  link  itself 
that  is  missing  or  the  nodes  that  were  to  be  connected. 

Third,  much  of  what  we  want  to  test  involves 
intermediate  processing.  That  is,  we  are  less  interested  in 
a  final  answer  than  in  the  steps  the  individual  took  to 
obtain  the  answer.  And  finally,  we  want  to  make  new  kinds 
of  inferences  based  upon  the  results  of  our  assessment.  We 
want  to  estimate  how  organized  is  the  individual's  knowledge 
of  a  domain  and  how  well  the  individual  can  use  the 
knowledge  he  or  she  has  acquired. 

Clearly  the  attempts  to  measure  schema  knowledge 
presented  in  this  paper  are  just  the  first  steps.  For  wider 
application  we  must  develop  new  concepts  of  validity  and 
reliability.  The  need  for  good  estimates  is  as  great  for 
tests  of  schema  knowledge  as  for  a  traditional  test.  One 
drawback  to  estimating  validity  is  that  we  do  not  have 
alternative  tests  that  measure  schema  acquisition.  At  this 
point  we  are  validating  the  items  by  comparing  them  with 
interview  responses  given  by  the  students  at  the  end  of  each 
i  ns t r uc t ional  session.  In  the  interview  we  have  the 
students  describe  what  they  have  studied,  have  them  define 
any  concepts,  have  them  demonstrate  the  ability  to  use  the 
different  parts  of  schema  knowledge,  and  have  them  talk 
generally  about  complex  story  problems.  We  expect  that 
students  who  have  difficulty  with  the  ICAI  system  will 
demonstrate  similar  difficulty  in  the  interviews. 
Comparisons  of  student  responses  to  the  computer  and 
responses  in  the  interview  are  now  underway. 


Many  issues  remain  to  be  resolved  in  future  research. 
Some  have  been  pointed  out  above.  Others  include  whether 
one  should  aggregate  test  item  scores  or  develop  a  profile 
of  knowledge.  How  would  the  aggregation  be  done?  How  would 
the  contributions  of  each  item  be  weighted?  What  advantages 
or  disadvantages  would  aggregation  have  over  the  profile? 
Similarly,  one  needs  to  decide  whether  to  develop  a 
quantitative  measure  for  each  item  or  whether  to  accept 
qualitative  scoring  on  some.  Again,  the  advantages  and 
disadvantages  need  to  be  studied.  Further, we  should  look  at 
the  amount  of  time  an  individual  takes  to  respond  to  items 
such  as  those  described  in  this  paper.  Would  reaction  times 
be  good  measures  of  access  or  retrieval?  Could  the  items  be 
modified  to  use  reaction-time  measures?  These  and  other 
related  questions  have  yet  to  be  studied. 
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Table  1 


Four  Students'  Responses  to 
the  ICAI  presentation  of  CHANGE  items 


GENERAL  PERFORMANCE 


STUDENT 
J  N  P 


N.imber 

of 

items  presented 

21 

22 

24 

Number 

of 

correct  responses 

18 

16 

15 

Per  cent 

cor  rect 

86% 

73% 

63% 

ANALYSIS  OF 

SCHEMA 

KNOWLEDGE 

STUDENT 


J 

N 

P 

COMPONENTS : 

Feature  Recognition: 

verbal 

66% 

50% 

50% 

visual 

66% 

50% 

80% 

Constraints : 

verbal 

100% 

100% 

100% 

visual 

100% 

75% 

72% 

CONNECTIVITY: 

Planning< — >Feature  Recognition 

75% 

66% 

50% 

Planning< — Constraints 

100% 

100% 

33% 

Schema  X 


Figure  4 


First  Feature  Recognition  Task  from  the  ICAI  System 


INSTRUCTIONS:  Read  the  story  below.  Decide  which  of  the 
five  situations  bast  describes  the  story  When  you  have 
made  your  choice,  position  the  arrow  on  top  of  the  one  you 
have  selected  and  click  the  mouse  button  once. 


The  History  Final  Exam  had  SO  questions.  There  were 
22  multiple  choice  items.  S  matching  exercises,  and 
3  essay  questions. 


Change 

Group 

Compare 

Vary 

Restate 


INSTRUCTIONS:  Choose  the  one  diagram  below  that  fits  this  story  problem.  Move 
the  arrow  into  the  diagram  you  have  selected  and  click  the  mouse  button. 


Bill  and  Eddie  both  planted  tomatoes  this  year.  Bill  harvested  twice  as  many  tomatoes 
as  Eddie.  Bill  picked  t4 tomatoes.  How  many  tomatoes  did  Eddie  have  ? 


Figure  6 

Constraints  Task  from  the  ICAI  System 


Figure  7 

Planning  Task  from  the  ICAI  System 
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